Summary of Changes

Hello @Jintao-Huang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on enhancing the gpt_bridge's compatibility with version 5 of the Hugging Face transformers library. The primary goal is to ensure accurate and robust handling of model state dictionaries, particularly for various Mixture of Experts (MoE) architectures, during conversions between different model formats. The changes involve refining the logic for identifying grouped experts and gate-up projections, making the conversion process more adaptable to the evolving structures of modern large language models.

Highlights

• Enhanced Transformers 5.0 Compatibility: Updated the gpt_bridge to improve compatibility with the transformers library version 5.0, specifically addressing how Mixture of Experts (MoE) models are handled during state dictionary conversions.
• Refactored MoE State Handling: Modified the _set_mlp_state method to dynamically determine hf_grouped and is_gate_up flags based on the conversion direction (to_mcore or to_hf) and the structure of the state dictionary or Hugging Face MLP module.
• Introduced Model-Specific Grouping Logic: Added a new helper method _get_hf_grouped to provide model-specific overrides for hf_grouped and is_gate_up for a range of MoE models (e.g., Qwen2-MoE, Deepseek-V2, GLM4-MoE), ensuring correct state mapping.
• Added Utility Imports: Included re and shutil modules, likely for advanced string pattern matching and file operations related to state dictionary processing.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.